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The  Value  of 


TRAINING: 


Analysis  of  DAU’s  Requirennents 
Managennent  Training 


RESULTS 


Charles  M.  Court,  Gregory  B.  Prothero,  and  Roy  L.  Wood 


In  response  to  Congress,  the  Defense  Acquisition  University  (DAU)  designed 
and  fielded  a  course  of  study  for  Requirements  Management,  including  a 
1-week  advanced  classroom  course.  While  teaching  this  course,  the  DAU 
faculty  routinely  conducts  pre-testing  and  post-testing  to  assist  the  faculty 
and  students  in  assessing  learning  and  retention.  The  faculty  uses  data  from 
these  tests,  along  with  student  demographics,  to  assess  the  value  of  learning 
the  course  provides  and  to  explore  some  initial  assumptions  about  the  readi¬ 
ness  of  the  workforce  to  learn.  Results  show  a  greater  than  30  percent  increase 
in  learning  from  pre-  to  post-test  and  debunk  nearly  all  the  preconceived 
notions  the  university  held  about  the  incoming  students. 
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Every  successful  system  acquisitionbegins  with  a  well-thought-out 
set  of  operational  capability  requirements.  The  military  services  have 
always  had  some  sort  of  requirements  generation  process  that  told  the 
armories  and  shipyards  what  to  build  for  the  warfighter.  As  acquisition 
became  more  complex,  expensive,  and  risky,  the  Department  of  Defense 
(DoD)  recognized  the  need  for  a  more  formal  system  of  articulating 
requirements  and  the  importance  of  training  both  the  acquisition  and 
the  requirements  workforces. 


The  Joint  Capabilities  Integration 
and  Development  System 

In  2003,  then-Secretary  of  Defense  Donald  Rumsfeld  initiated  a  for¬ 
mal  DoD-level  requirements  generation  process— the  Joint  Capabilities 
Integration  and  Development  System  (JCIDS).  According  to  Chairman 
of  the  Joint  Chiefs  of  Staff  Instruction  (CJCSI)  3170.01H,  “The  JCIDS 
process  exists  to  support  JROC  [Joint  Requirements  Oversight  Council] 
and  CJCS  [Chairman  of  the  Joint  Chiefs  of  Staff]  responsibilities  in 
identifying,  assessing,  validating,  and  prioritizing  joint  military  capa¬ 
bility  requirements”  (CJCS,  2012).  Within  the  context  of  the  National 
Military  Strategy,  JCIDS  provides  a  process  to  identify  and  assess  the 
capabilities  joint  operational  forces  need  to  meet  future  military  chal¬ 
lenges.  A  capabilities-based  assessment  process  identifies  potential  gaps 
in  warfighting  capability  and  drives  changes  to  doctrine,  organization, 
training,  materiel,  leadership  and  education,  personnel,  facilities,  and/or 
policy  (DOTmLPF-P).  Many  requirements  lead  to  nonmateriel  solutions, 
while  other  requirements  call  for  materiel  solutions.  The  JCIDS  process 
generates  the  requirements  and  the  associated  performance  criteria  for 
those  materiel  solutions.  The  Defense  Acquisition  Management  System 
then  fulfills  those  requirements  and  delivers  the  required  capabilities. 

Articulating  a  new  warfighting  capability  requirement  and  defending 
this  need  through  rigorous  discussion  and  analysis  is  a  nontrivial  under¬ 
taking  for  a  requirements  manager.  A  new  military  requirement  can 
initiate  a  decades-long  acquisition  that  requires  the  investment  of  bil¬ 
lions  of  taxpayer  dollars  to  develop,  manufacture,  and  field.  Requirements 
managers  must  be  able  to  correctly  identify,  document,  and  support  the 
compelling  need  for  any  new  system,  then  be  able  to  work  alongside  their 
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acquisition  counterparts  to  field  the  new  capability.  This  is  a  complex 
undertaking.  In  2007,  Congress  formally  directed  the  DoD  to  train  the 
men  and  women  who  develop  new  requirements  under  JCIDS. 


mately  validate  the  requirements.  This 
mandate  created  a  need  for  a  broad  and 
diverse  training  program  at  several  lev¬ 
els  of  sophistication.  Further,  as  Court 
(2010)  pointed  out,  “no  one  person  does 
all  four  tasks  of  assessing,  developing, 
validating,  and  prioritizing”  require¬ 
ments,  so  the  training  program  would 
also  need  to  address  a  wide  variety  of 
tasks  and  competencies. 


Requirements  Management  Training 

The  National  Defense  Authorization  Act  (NDAA)  of  2007  man¬ 
dated  the  Under  Secretary  of  Defense  for  Acquisition,  Technology,  and 
Logistics  (AT&L),  in  consultation  with 
the  Defense  Acquisition  University 
(DAU),  to  develop  a  training  program 
to  certify  DoD  personnel  with  the 
responsibility  to  generate  capabil¬ 
ity  requirements  for  major  defense 
acquisition  programs  (NDAA,  2006). 

The  congressional  mandate  called  for 
training  both  military  and  DoD  civil¬ 
ian  managers  charged  with  assessing, 
developing,  validating,  and  prioritiz¬ 
ing  requirements  through  the  JCIDS 
process.  This  broad  definition  covered 
relatively  junior  members  of  the  work¬ 
force  up  to  and  including  4-star  generals 
and  admirals  on  the  JROC  who  ulti¬ 


DAU  responded  quickly  to  meet  the  congressionally  imposed  deadline 
to  create  and  deploy  a  requirements  management  certification-training 
curriculum  by  September  30,  2008.  Working  with  AT&L  and  the  Joint 
Staff  Directorate  for  Force  Structure,  Resources  and  Assessment  (J8), 
DAU  developed  two  online  courses  for  requirements  managers  and  a 
1-day  classroom  workshop  for  general  and  flag  officers.  These  courses 
were  very  successful,  and  by  the  end  of  fiscal  year  2008,  the  community 
had  logged  more  than  4,200  course  completions.  In  2010,  DAU  added  a 
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1-week  Advanced  Concepts  and  Skills  for  Requirements  Management 
(RQM  310)  classroom  capstone  course  to  the  curriculum.  Table  1  shows 
the  requirements  management  curriculum  for  designated  individuals 
as  recently  as  2014. 


Requirements  Management  Training  Curricuium 

Developing  new  courses  for  requirements  management  was  an 
entirely  new  area  for  DAU  training  outside  the  customary  acquisition 
disciplines.  The  effort  demanded  an  intense  effort  from  DAU, 
supported  and  sponsored  by  both  the  AT&L  staff  and 
the  Joint  Staff  DAU  established  integrated  product 
teams  that  included  warfighter  representa¬ 
tives  to  define  the  basic  competencies 
requirements  managers  need  to  oper¬ 
ate  successfully  at  different  levels 
of  responsibility.  The  DAU 
faculty  and  outside  subject 
matter  experts  meticu- 
lously  developed  instruction  to 
meet  these  competencies  across 
the  spectrum  of  requirements 
tasks.  The  faculty  adopted  several 
innovative  assessment  tools  to  help 
DAU  answer  the  question  of  whether 
or  not  the  training,  once  deployed,  would  be 
effective. 


Requirements  Certification  Capstone  Course:  New 
Beginnings  and  Opportunities 

Developing  RQM  310,  the  Advanced  Concepts  and  Skills  for 
Requirements  Management  course  demanded  an  intense,  months-long 
effort  by  requirements  and  acquisition  experts  to  ensure  the  course 
conformed  to  the  requirements  management  competency  model  and 
would  challenge  students  to  reach  higher  levels  of  understanding  and 
performance.  DAU  designed  and  piloted  the  new  1-week  course  and 
rolled  it  out  to  students  in  2010. 

Creating  an  entirely  new  classroom  course  allowed  DAU  to  test  and 
apply  many  new  concepts  and  technologies.  RQM  310  includes  faculty 
discussions,  guest  speakers,  computer  simulations,  and  a  challenging 
student  capstone  exercise.  One  of  the  technology  innovations  in  RQM 
310  was  the  routine  use  of  a  classroom-participation  system.  With 
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this  system,  each  student  uses  a  response  device  that  looks  like  a  small 
remote  control  to  respond  to  questions  and  assessments.  During  the  first 
morning  of  the  class,  students  use  their  response  devices  to  take  a  course 
pre-test  and  review  material  from  the  course’s  online  prerequisites. 
Throughout  the  week,  students  continue  to  use  the  response  device  to 
interact  with  faculty  questions  in  the  lessons.  The  RQM  310  students 
also  use  the  response  devices  in  an  in-class  simulation  to  evaluate  and 
discuss  differences  between  programs  depending  on  their  timeline, 
financial  state.  Service  and  Defense  Agency  priorities,  and  issues  such 
as  a  budget  breach  or  a  failed  operational  test. 

RQM  310  student  demographics.  Both  military  and  civilian  require¬ 
ments  managers  attend  RQM  310.  Students  come  from  the  Pentagon 
as  well  as  from  far-flung  Combatant  Commands  and  field  activities. 
Military  members  bring  current  and  relevant  experience  to  the  require¬ 
ments  generation  process.  Typically,  military  requirements  managers 
come  from  operational  and  warfighting  specialties,  and  complete  a 
requirements  management  tour  between  field  assignments.  However, 
there  is  a  relatively  high  turnover  of  military  personnel  through  require¬ 
ments  management  positions,  bringing  in  new  personnel  with  limited  to 
no  JCIDS  or  acquisition  experience,  thus  creating  a  steady  demand  for 
training.  Civilian  requirements  managers  have  greater  tenure  in  their 
positions,  and  provide  continuity  in  requirements  offices  and  a  “corpo¬ 
rate  memory”  for  their  organizations. 

Assumptions  about  the  workforce.  Given  the  vastly  different  demo¬ 
graphics  of  the  workforce  who  attend  RQM  310,  initial  expectations  were 
that  incoming  knowledge  and  experience  of  the  students  might  also  be 
vastly  different.  For  example,  the  DAU  faculty  assumed  that  civilian 
requirements  managers,  because  of  their  longer  tenure,  would  be  better 
versed  in  JCIDS  and  acquisition  procedures  than  their  military  coun¬ 
terparts.  Another  commonly  held  belief  was  that  students  working  in 
the  nation’s  capital  or  on  a  combatant  commander’s  staff  would  be  more 
knowledgeable  coming  into  the  course  because  of  more  direct  involve¬ 
ment  in  generating  and  vetting  requirements.  In  addition  to  assessing 
the  overall  value  of  training,  this  study  tested  these  major  assumptions 
about  the  workforce,  and  the  results  are  presented  later  in  this  article. 
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Study  Method 

Participants 

This  study  used  the  data  the  DAU  faculty  normally  collects  in  the 
process  of  executing  each  RQM  310  class.  For  purposes  of  this  study,  the 
data  collected  were  from  the  2013  course  offering.  The  faculty  did  not 
originally  anticipate  using  this  course  pre-test  data  in  a  study,  but  rather 
as  a  review  specifically  to  assist  the  students  in  identifying  their  own 
individual  knowledge  gaps,  and  to  alert  the  faculty  to  particular  areas 
of  knowledge  weakness  in  the  class  as  a  whole.  Educational  research  has 
consistently  shown  that  pre-testing  can  help  increase  student  attentive¬ 
ness  during  the  course  (Sadhasivam,  2013),  and  aid  in  focusing  both 
students  and  faculty  on  improvement  of  particular  knowledge  gaps  (Blin 
&  Wilson,  1994;  Wetstein,  1998). 

While  DAU  developed  the  assessments  and  data  collection  primarily 
to  improve  learning  outcomes,  the  data  have  been  useful  in  providing 
valuable  insights  into  other  aspects  of  the  training.  The  DAU  faculty 
compares  pre-test  data  to  post-test  data  to  determine  overall  student 
improvement  and  to  assess  the  value  of  learning.  Post-test  data  from 
the  end-of-course  assessment  have  similar,  but  not  identical,  questions 
as  those  on  the  pre-test.  The  faculty  also  analyzed  pre-test  data  in  this 
study  against  student  demographics  to  determine  whether  one  group 
might  be  better  prepared  for  the  advanced  concepts  course. 


The  DAU  faculty  compares  pre-test  data  to 
post-test  data  to  determine  overall  student 
improvement  and  to  assess  the  value  of  learning. 
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Research  Design 

As  noted  earlier,  this  research  used  data  collected  from  a  total  of 
263  students  during  the  normal  execution  of  the  RQM  310  course  in 
2013.  The  data  collected  include  pre-test  and  end-of-course  assess¬ 
ment  scores  collected  with  the  student  response  system.  Questions  on 
the  two  tests  are  similar,  but  not  identical,  and  both  instruments  focus 
on  key  learning  and  competencies  needed  by  requirements  managers 
to  be  effective  in  their  jobs.  All  of  the  students  attending  the  RQM  310 
advanced  course  had  previously  completed  the  two  online  prerequi¬ 
site  courses;  Introduction  to  the  Joint  Capabilities  Integration  and 
Development  System  (CLR 101),  and  Core  Concepts  for  Requirements 
Management  (RQM  110).  These  online  courses  are  self-paced,  computer- 
based  training  that  include  their  own  online  assessments  of  student 
progress  and  understanding.  RQM  110  classes  have  assigned  faculty 
who  are  available  to  answer  questions,  mentor  students  who  might  be 
experiencing  difficulty  in  the  course,  and  otherwise  provide  academic 
or  technical  assistance  the  students  might  need. 

DAU  also  collects  student  demographics  in  the  RQM  310  class  to  help  the 
faculty  better  appreciate  the  level  of  experience  and  exposure  to  identi¬ 
fying,  assessing,  and  formulating  capability  requirements.  Based  on  a 
priori  assumptions  mentioned  earlier,  the  faculty  collects  student  data 
on  each  student’s  assignment  at  the  time  he  or  she  attended  the  course, 
their  tenure  in  their  current  billet,  aggregate  experience  working  in  the 
requirements  management  field,  and  how  much  of  each  student’s  day- 
to-day  work  content  related  to  managing  requirements.  Table  2  shows 
a  breakdown  of  the  demographic  questions  and  the  granularity  of  the 
answers  collected. 

Analysis  of  Pre-Test  and  Post-Test  Scores 

As  a  first  step  in  this  analysis,  tabulating  and  analyzing  pre-test 
scores  produced  a  mean  score  of  51.6  with  standard  deviation  (s.d.)  of 
12.81.  The  tally  of  end-of-course  scores  showed  substantial  improvement 
with  a  mean  of  80.97  and  s.d.  of  10.68.  A  paired-samples  t-test  showed 
the  improvement  in  scores  to  be  statistically  significant,  t(262)  =  37.173, 
p  <  0.0005.  As  noted  earlier,  many  researchers— and  faculty  practitio¬ 
ners— recognize  that  pre-testing  students  can  help  focus  their  attention 
on  desired  outcomes  and  infiuence  post-test  outcomes.  According  to 
Kim  and  Wilson  (2010),  “there  can  be  substantial  effects  of  pretest  on 
posttest,  especially  when  the  duration  between  them  is  short,  that  is,  less 
than  a  month”  (p.  755).  Researchers  must  consider  and  compensate  for 
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this  fact  in  a  strict  research  context.  However,  since  the  underlying  pur¬ 
pose  of  the  classes  was  to  improve  student  knowledge  and  retention,  the 
substantial  improvement  in  scores  was  desirable  regardless  of  the  cause. 


TABLE  2. 

STUDENT  DEMOGRAPHIC  CATEGORIES 

Beltway? 

Time 
in  Billet 

Experience 

Percent 

Requirements 

Work 

Career  Field 

Organization 

Inside 

0-6 

0-6 

0-25% 

Require¬ 

Joint  Staff 

Months 

Months 

ments 

Outside 

6-12 

6-12 

25-50% 

Operations 

Service  HQ 

Months 

Months 

Staff 

12-24 

1-3  Years 

50-75% 

Acquisition 

Major 

Months 

Command 

>  24 

3-5  Years 

75-100% 

Other 

Defense 

Months 

Agency 

>  5  Years 

100% 

OSD  Staff 

Other 

Analysis  of  the  Student  Demographics 

As  noted  earlier,  a  number  of  assumptions  about  the  student  demo¬ 
graphics  produced  expectations  among  faculty  for  those  who  might 
perform  better  in  the  class,  and  those  who  might  require  more  assistance 
or  remediation.  During  this  research  process,  the  DAU  faculty  wanted 
to  test  these  assumptions  statistically  to  determine  their  accuracy.  To 
do  so,  the  faculty  tested  each  of  the  assumptions  using  SPSS  t-tests  or 
analysis  of  variation  (ANOVA)  to  examine  the  mean  scores  of  each 
subgroup  on  the  pre-test  data.  The  discussion  below  outlines  the  assump¬ 
tions  and  test  results.  In  short,  almost  none  of  the  entering  assumptions 
proved  to  be  true,  and  the  classes  were  far  more  homogeneous  in  terms  of 
pre-test  performance  without  regard  to  prior  experience  or  assignment. 

Assumption  1.  Students  from  inside  the  (Washington,  DC)  Beltway 
would  be  better  prepared  than  those  in  field  activities  outside  the 
Beltway.  An  independent-samples  t-test  assessed  the  means  of  the 
pre-test  scores  between  the  two  groups.  The  inside-the-Beltway  group 
average  pre-test  score  was  52.28  ±  12.5  and  the  outside-the-Beltway 
group  posted  an  average  score  of  59.79  ±13.2.  The  t-test  analysis  found 
no  statistically  significant  differences  between  student  groups  at  a  95% 
confidence  level,  t(263)  =  0.93, p  =  0.473. 
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Assumption  2.  Students  with  more  time  in  their  current  billet  will 
be  better  prepared  than  those  with  shorter  tenures.  The  assessment 
divided  the  students  into  those  with  less  than  6  months  in  their  current 
positions,  those  with  6-12  months,  those  with  12-24  months’  tenure, 
and  those  with  greater  than  24  months  in  the  job.  Since  many  military 
requirements  managers  historically  have  shorter  tours  in  requirements 
billets  between  operational  tours,  observers  could  assume  that  longer 
tenures  might  better  prepare  students  for  the  advanced  course.  The 
analysis  did  not  support  this  assumption,  however.  The  means  of  the 
group  scores  on  the  pre-test  varied  only  between  49.5  and  53.7.  An 
ANOVA  test  on  the  groups  revealed  no  statistically  significant  differ¬ 
ences  in  their  respective  performances  on  the  pre-test,  F(3,  258)  =  1.11, 
p  =  0.344. 


Assumption  3.  Students  with  greater  experience  in  requirements  man¬ 
agement  would  be  better  prepared.  To  test  this  assumption,  the  analysis 
subdivided  the  students  into  groups  with  less  than  6  months'  experi¬ 
ence,  those  with  6-12  months'  tenure,  1-3  years,  3-5  years,  and  greater 
than  5  years.  An  ANOVA  test  on  this  data  did  find  a  single  statistically 
significant  difference  between  groups  of  students  as  determined  by  the 
one-way  ANOVA,  F(4,  258)  =  3.096,  p  =  0.016.  A  Tukey  post-hoc  test  on 
the  data  revealed  that  students  with  3-5  years  of  experience  showed  a 
statistically  significant  average  higher  score  (56.7  versus  48.2)  on  the 
pre-test  than  less  experienced  students  with  6-12  months'  experience. 

Assumption  4.  Students  who  spend  a  greater  amount  of  day-to-day  time 
working  on  requirements  will  show  better  preparation  for  the  class.  For 
this  test,  the  analysis  divided  the  students  into  five  groups;  (1)  students 
who  reported  working  on  requirements-related  tasks  less  than  25%  of 
the  time;  (2)  those  with  requirements  work  between  25%  and  50%;  (3) 
students  with  requirements  work  from  50%  to  75%;  (4)  those  whose 
requirements  content  in  their  workday  were  between  75%  and  100%;  (5) 
students  whose  work  was  100%  exclusively  related  to  requirements.  The 
ANOVA  analysis  for  these  groups  again  pointed  to  no  statistical  differ¬ 
ences  between  the  pre-test  means,  1^(5,257)  =  1.48,  p  =  0.195.  The  pre-test 
average  scores  for  these  groups  varied  only  between  50  and  53.6. 

Assumption  5.  Designated  requirements  managers,  and  perhaps  acqui¬ 
sition  professionals,  will  be  better  prepared  for  the  class.  Here,  the 
demographic  questions  asked  the  students  to  self- identify  their  primary 
career  field:  requirements,  acquisition,  operational/warfighter,  or  other. 
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The  ANOVA  analysis  of  the  mean  pre-tests  scores  for  these  groups  found 
no  statistically  significant  differences,  with  mean  scores  between  48.9 
and  53.6,  F(3, 259)  =  0.880,  p  =  0.452. 

Assumption  6.  Organizational  assignment  will  have  some  impact  on 
student  readiness.  The  initial  assumption  was  that  there  might  be  some 
relationship  between  the  student’s  assigned  organization  and  his  or  her 
score  on  the  pre-test.  For  example,  the  faculty  might  expect  a  student 
assigned  to  the  Joint  Staff  or  Combatant  Command  to  do  more  work 


This  analysis  debunked  nearly  every  assumption 
about  factors  that  might  affect  student 
preparedness  for  the  advanced  course. 


directly  or  indirectly  in  creating,  assessing,  or  approving  requirements 
than  students  from  other  organizations.  For  this  analysis,  the  study 
broke  the  student  sample  into  those  who  worked  on  the  Joint  Staff, 
Service  Headquarters  Staff,  major  military  command.  Defense  Agency, 
Cffice  of  the  Secretary  of  Defense  Staff,  a  Combatant  Commander  Staff, 
or  other.  Cnee  again,  the  ANCVA  showed  no  statistical  differences  in 
mean  pre-test  scores  of  the  students,  regardless  of  their  assignment,  F(6, 
256)  =  0.312,p  =  0.930. 

Significance  of  the  Anaiysis 

This  analysis  debunked  nearly  every  assumption  about  factors  that 
might  affect  student  preparedness  for  the  advanced  course.  Each  of  these 
assumptions  made  sense  on  an  intuitive  level,  and  the  results  have  been 
surprising.  DAU  will  need  to  do  more  work  to  determine  exactly  why 
these  assumptions  were  untrue,  but  preliminary  analysis  offers  two 
potential  explanations.  First,  the  knowledge  of  students  coming  into 
the  course  is  much  more  homogeneous  than  originally  believed.  This 
may  be  the  result  of  all  students  being  required  to  take  the  same  online 
preparatory  courses.  Introduction  to  the  Joint  Capabilities  Integration 
and  Development  System,  CLR 101,  and  Core  Concepts  for  Requirements 
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Management,  RQM 110.  Students  who  take  these  courses  may  come  into 
the  advanced  RQM  310  with  a  common  baseline  of  knowledge  learned 
primarily  from  those  classes.  Another  possibility  is  that  individuals  in 
the  requirements  community  typically  work  only  on  single  or  perhaps 
a  handful  of  tasks  related  to  the  broader  process  of  identifying,  assess¬ 
ing,  validating,  and  prioritizing  joint  requirements.  It  is  unlikely  that 
any  individual  student  would  have  a  deep  knowledge,  based  on  experi¬ 
ence,  across  the  entire  process,  regardless  of  tenure  or  organizational 
assignment.  Thus,  expertise  in  any  narrow  area  may  not  contribute  to 
statistically  higher  scores  on  course  material  that  covers  all  areas. 


Summary  and  Conclusions 

DAU  responded  to  the  congressional  mandate  and  met  the  short 
deadline  to  train  and  certify  requirements  managers  through  a  com¬ 
bination  of  online  and  classroom  courses.  The  success  of  the  initial 
DAU  approach  led  to  student  demand  and  leadership  support  to  expand 
the  initial  requirements  curriculum.  The  most  significant  curriculum 
expansion  was  the  development  of  the  Advanced  Concepts  and  Skills  for 
Requirements  Management  course,  RQM  310. 

Developing  a  new  classroom  course  in  a  different,  nontraditional  area 
of  acquisition  allowed  the  DAU  faculty  to  apply  new  technologies. 
Classroom  simulations  enhanced  traditional  teaching  approaches.  The 
simulations  encouraged  the  exchange  of  ideas.  They  helped  requirements 


This  analysis  has  also  been  a  “myth  buster” for 
a  number  of  sincerely  held  assumptions  about 
the  workforce  and  how  demographic  factors 
influence  RQM  310  student  preparation. 
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managers  from  different  Services  and  Defense  Agencies  recognize  their 
common  problems.  Classroom  participation  devices  encouraged  more 
student  involvement. 

The  success  of  using  classroom-participation  devices  led  the  require¬ 
ments  faculty  to  additional  innovation.  Students  take  a  pre-test  on  the 
first  day  of  class,  and  a  final  exam  post-test  at  the  end  of  the  1-week 
course.  Both  exams  use  classroom-participation  “clickers”  with  the 
exam  questions  projected  on  a  classroom  screen.  By  comparing  the 
results  of  the  pre-test  to  the  results  of  the  post-test,  this  analysis 
has  established  that  statistically  significant  improvements  in  scores 
occur,  leading  us  to  conclude  with  confidence  that  student  learning  was 
taking  place. 

This  analysis  has  also  been  a  “myth  buster”  for  a  number  of  sincerely 
held  assumptions  about  the  workforce  and  how  demographic  fac¬ 
tors  infiuence  RQM  310  student  preparation.  Almost  universally,  the 
assumptions  have  been  wrong,  and  students  coming  into  the  course 
are  much  more  homogeneous  than  the  faculty  anticipated.  Part  of  the 
homogeneity  could  result  from  all  students  taking  the  same  prerequisite 
courses— CLR 101  and  RQM  110— and  coming  into  the  advanced  RQM 
310  with  a  common  baseline  of  knowledge  learned  from  those  classes. 
Another  possibility  is  that  individuals  in  the  community  work  only 
on  single  or  perhaps  a  handful  of  tasks  related  to  identifying,  assess¬ 
ing,  validating,  and  prioritizing  joint  requirements,  thus  no  individual 
student  has  a  deep  knowledge  across  the  entire  process,  regardless  of 
tenure  or  organizational  assignment.  Expertise  in  a  narrow  area  may 
not  contribute  to  statistically  higher  scores  on  course  material  that 
covers  all  areas. 

Nevertheless,  the  success  of  pre-  and  post-testing  in  RQM  310  has 
encouraged  the  faculty  to  expand  this  approach  to  other  requirements 
courses.  Specifically,  the  faculty  is  investigating  how  to  apply  this 
approach  to  the  online  Core  Concepts  for  Requirements  Management 
course,  RQM  110.  Further,  based  on  the  success  of  RQM  310,  additional 
classroom  courses  at  the  Defense  Systems  Management  College  have 
adopted  the  classroom  simulations  and  the  student-participation  sys¬ 
tem,  and  are  collecting  student  demographics  and  learning  data  to  be 
able  to  continuously  improve  course  content  and  learner  performance. 
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Research  Limitations  and  Future 
Research 

As  noted  earlier,  the  data  collected  from  the  RQM  students  were  pri¬ 
marily  for  the  purpose  of  gauging  the  knowledge  of  the  incoming  students 
and  ensuring  that  the  course  delivered  important  content  in  a  way  that 
was  understandable  and  memorable.  This  analysis  did  not  use  random 
samples  or  experimental  methods  that  would  contribute  to  a  rigorous 
scientific  study.  Future  researchers  may  choose  to  close  these  obvious 
gaps  in  a  more  intentional  way.  In  addition,  post-testing  performed  at 
the  end  of  the  class  does  not  guarantee  the  students  will  remember  the 
information  over  the  long  term.  Future  research  may  wish  to  test  stu¬ 
dents  several  weeks  or  months  after  graduation  and  assess  the  results 
of  knowledge  retention  over  time. 
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Appendix 

The  ROM  310  Class  Schedule 

Table  A1  illustrates  when  the  DAU  faculty  administers  the  pre¬ 
course  assessment  and  the  end-of-course  examination.  The  table  also 
lists  the  course  topics  and  uses  a  color  code  to  illustrate  the  different 
class  activities.  Table  A2  explains  the  color  code. 


TABLE  A1.  ROM  310  DAILY  CLASS  SCHEDULE 


8:00 


8:30 


9:00 

9:30 

10:00 


10:30 

11:00 

11:30 

12:00 


12:30 

13:00 


13:30 


14:00 

14:30 


Monday  Tuesday  Wednesday  Thursday  Friday 
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15:00 

JCIDS 

Simulation 

Examination 

Retest 

15:30 

ICD  Review 
Exercise 

Milestone  A  to 
Milestone  B 

16:00 

SIM  Debrief 

16:30 

Test 

Questions 

Examination 

Results 

17:00 

Note.  AoA  =  Analysis  of  Alternatives;  CDD  =  Capability  Development  Document; 
DOTmLPF-P  =  Doctrine,  Organization,  Training,  Materiel,  Leadership  and  Education, 
Personnel,  Facilities,  and  Policy;  FCB  =  Functional  Capabilities  Board;  FOC  = 

Full  Operational  Capability;  ICD  =  Initial  Capabilities  Document;  IT  =  Information 
Technology;  IS  =  Information  System;  JCIDS  =  Joint  Capabilities  Integration  and 
Development  System;  KPP  =  Key  Performance  Parameter;  KSA  =  Key  System 
Attribute;  MDD  =  Materiel  Development  Decision;  PPBE  =  Planning,  Programming, 
Budgeting,  and  Execution;  ROM  =  Requirements;  SIM  =  Simulation. 
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